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Abstract. Deciding in an efficient way weak probabilistic bisimulation in the context of prob- 
abilistic automata is an open problem for about a decade. In this work we close this problem by 
proposing a procedure that checks in polynomial time the existence of a weak combined transi- 
tion satisfying the step condition of the bisimulation. This enables us to arrive at a polynomial 
time algorithm for deciding weak probabilistic bisimulation. We also present several extensions 
to interesting related problems setting the ground for the development of more effective and 
compositional analysis algorithms for probabilistic systems. 

1 Introduction 

Probabilistic automata (PA) constitute a mathematical framework for the specification of proba- 
bilistic concurrent systems HE]. Probabilistic automata extend classical concurrency models in 
a simple yet conservative fashion. In probabilistic automata, there is no global notion of time, and 
probabilistic experiments can be performed inside a transition. This embodies a clear separation be- 
tween probability and nondeterminism, and is represented by transitions of the form s /j, where 
s is a state, a is an action label, and /i is a probability distribution on states. Labeled transition sys- 
tems are instances of this model family, obtained by restricting to Dirac distributions (assigning full 
probability to single states). Thus, foundational concepts and results of standard concurrency theory 
are retained in full and extend smoothly to the model of probabilistic automata. The PA model is 
akin to Markov decision processes (MDP) [7|, and its foundational beauty can be paired with pow- 
erful model checking techniques, as implemented for instance in the PRISM tool 1731 . Variations of 
this model are Labeled Concurrent Markov Chains (LCMQ and alternating Models II11II20II261 . We 
refer the interested reader to [E2l for a survey on PA and other models. 

If facing a concrete probabilistic system, we can conceive several different PA models to re- 
flect its behavior. For instance, we can use different state names, encode diverse information in the 
states, represent internal computations with different action labels, and so on. Bisimulation relations 
constitute a powerful tool allowing us to check whether two models describe essentially the same 
system. They are then called bisimilar. The bisimilarity of two systems can be viewed in terms of a 
game played between a challenger and a defender. In each step of the infinite bisimulation game, the 
challenger chooses one automaton, makes a step, and the defender matches it with a step of the other 
automaton. Depending on how we want to treat internal computations, this leads to strong and weak 
bisimulations: the former requires that each single step of the challenger automaton is matched by an 
equally labeled single step of the defender automaton, the latter allows the matching up to internal 
computation steps. On the other hand, depending on how nondeterminism is resolved, probabilis- 
tic bisimulation can be varied by allowing the defender to match the challenger's step by a convex 
combination of enabled probabilistic transitions. This results in a spectrum of four bisimulations: 
strong [ 1 1 21 26], strong probabilistic lED . weak [20.21 1, and weak probabilistic lETI bisimulation. 

Besides comparing automata, bisimulation relations allow us to reduce the size of an automaton 
without changing its properties (i.e., with respect to logic formulae satisfied by it). This is particularly 
useful to alleviate the state explosion problem notoriously encountered in model checking. 

Polynomial decision algorithms for strong (probabilistic ) bisimulation O and weak bisimulation 
||20| are known. However, PA weak bisimulation lacks in transitivity and this severely limits its 
usefulness. On the other hand weak probabilistic bisimulation is indeed transitive, while the only 
known algorithm for such bisimulation is exponential [ 3 1 in the size of the probabilistic automaton. 



In this context, it is worth to note that LCMC weak bisimulation [20] and PA weak probabilistic 
bisimulation ETTl coincide ll23ll when LCMC is seen as a PA with restrictions on the structure of the 
automaton and that restricted versions of PA weak probabilistic bisimulations, such as normed fl] 
and delay [24] bisimulation, can be decided in polynomial time. Following [23 1, an LCMC is just a 
PA where each state with outgoing transitions enables either labeled transitions each one leading to 
a single state, or a single transition leading to a probability distribution over states and this constraint 
on the structure of the automaton is enough to reduce the complexity of the decision procedure at the 
expense of the loss of using combined transitions and nondeterminism to simplify the automaton. 

Lately, the model of PA has been enhanced with memoryless continuous time, integrated into 
the model of Markov automata (6][8][9]. This extension is also rooted in interactive Markov chains 
(IMC) [13], another model with a well-understood compositional theory. IMCs are applied in a large 
spectrum of practical applications, ranging from networked hardware on chips to water treat- 
ment facilities lfl2l and ultra-modern satellite designs ifTOl . The standard analysis trajectory for IMC 
revolves around compositional applications of weak bisimulation minimization, a strategy that has 
been proven very effective ||2][5][l4|, an d is based on a polynomial time weak bisimulation deci- 
sion algorithm [ 13 27 ]. Owed to the unavailability of effective algorithms for PA weak probabilistic 
bisimulations, this compositional minimization strategy has thus far not been applied in the PA (or 
MDP) setting. We aim at making this possible, and furthermore, we intend to repeat and extend the 
successful applications of IMC in the extended Markov automata setting. For this, a polynomial time 
decision procedures for weak probabilistic bisimulation on PA is the essential building block. 

In this paper we show that PA weak probabilistic bisimulation can be decided in polynomial 
time, thus just as all other bisimulations on PA. To arrive there, we provide a decision procedure 
that follows the standard partition refinement approach [3. 16,181 and that is based on a Linear Pro- 
gramming (LP) problem. The crucial step is that we manage to generate and decide an LP problem 
that proves or disproves the existence of a weak step in time polynomial in the size of an automaton 
which in turn encodes a weak transition linear in its size. This enables us to decide in polynomial 
time whether the defender has a matching weak transition step - opposed to the exponential time 
required thus far fl3] for this. Apart from this result, which closes successfully the open problem 
of O, we show how our LP approach can be extended to hyper-transitions (weak transitions leaving 
a probability distribution instead of a single state) and to the novel concepts of allowed weak/hyper- 
transitions (weak/hyper-transitions involving only a restricted set of transitions) and of equivalence 
matching (given two states, check whether each one enables a weak transition matchable by the 
other). Hyper-transitions naturally occur in weak probabilistic bisimulation on Markov automata, 
and in the bisimulation formulation of probabilistic forward simulation ll8l l2"TI . 
Organization of the paper. After the preliminaries in Section|2] we present in Section|3]the polyno- 
mial LP problem that models weak transitions together with several extensions that can be computed 
in polynomial time as well. Then, in Section|U we recast the algorithm proposed in [3 1 that decides 
whether two probabilistic automata are weak probabilistic bisimilar and we show that the decision 
procedure is polynomial. We conclude the paper in Section [5] with some remarks, followed by ap- 
pendixes containing all detailed proofs. 

2 Mathematical Preliminaries 

For a generic set X, denote by Disc (AT) the set of discrete probability distributions over X, and by 
SubDisc(A) the set of discrete sub-probability distributions over X. Given p e SubDisc(A), we 
denote by Supp(p) the set {x 6 X \ p(x) > 0}, by p(_L) the value 1— p( X) where _L ^ X, and by 5 X 
the Dime distribution such that p(x) = 1 for x £ X U {^}. For a sub-probability distribution p, we 
also write p = {p x x \ x E X, p x = p(x)}. The lifting C(TL) IT7l of a relation TvLC X x Y is defined 
as follows: for px € Disc(A) and py € Disc(Y), px £(7£) Py holds if there exists a weighting 
function w: X x Y [0, 1] such that (l) w(x, y) > implies x 1Z y, (2) YlyeY w ( x i v) — Px(x), 
and (3) J2xex w i x i y) = Py(v)- When 1Z is an equivalence relation on a set X, p\ C(JZ) pi holds 
if foreach C G X/K, p x {C) = p 2 {C). 



A Probabilistic Automaton (PA) A is a tuple (S, s, £, D), where S is a set of states, s 6 S is the 
storf sfafe, 17 is the set of actions, and D <Z S x S x Disc(S') is a probabilistic transition relation. 
The set 17 is parted in two sets H and E of internal (hidden) and external actions, respectively; we 
let s,t,u,v, and their variants with indices range over S, a, b range over actions, and r range over 
hidden actions. In this work we consider only finite PAs, i.e., automata such that S and D are finite. 

A transition tr = (s, a, /i) € D, also denoted by s — — >• /i, is said to Zeave from state s, to be 
labeled by a, and to lead to /Lt, also denoted by \i tr . We denote by src(tr) the source state s, by 
act(tr) the action a, and by trg(tr) the target distribution /i. We also say that s enables action a, 
that action a is enabled from s, and that (s, a, /x) is enabled from s. Finally, we denote by D(s) the 
set of transitions enabled from s, i.e., D(s) = {tr G D | src(tr) = s}, and similarly by D(a) the 
set of transitions with action a, i.e., D(a) = {tr G D | act(tr) = a}. 

An execution fragment of a PA .4 is a finite or infinite sequence of alternating states and actions 
a = SQa\S\a2S2 ■ ■ ■ starting from a state sq, also denoted by first(a), and, if the sequence is finite, 
ending with a state, such that for each i > there exists a transition (sj_i, dj, /U.j) £ £ such that 
A*i(sj) > 0. If the sequence a is finite, then denote by last (a) the last state of a. The length of a, 
denoted by |a|, is the number of occurrences of actions in a. If a is infinite, then \a\ — oo. Denote by 
frags (A) the set of execution fragments of A and by frags* (A) the set of finite execution fragments 
of A. An execution fragment a is a prefix of an execution fragment a', denoted by a ^ a', if the 
sequence a is a prefix of the sequence a'. The trace of a, denoted by trace(a), is the sub-sequence 
of external actions of a. For instance, for a G E, trace(soasi) = trace(sQTSiT . . . TS n _i<zs n ) = a, 
also denoted by frace(a), and trace(so) = trace(s^TS\T . . .rs„) = e, the empty sequence, also 
denoted by trace(r). 

A scheduler for a PA ^4 is a function cr: frags* (A) — > SubDisc(D) such that for each finite 
execution fragment a, o~{a) G SubDisc(Z) (last (a))). A scheduler is determinate [3] if for each 
pair of execution fragments a, a', if trace (a) = <race(a') and last(a) — last (a'), then ct(q;) = 
cr(a'). Given a scheduler cr and a finite execution fragment a, the distribution c(a) describes how 
transitions are chosen to move on from last(a). A scheduler a and a state s induce a probability 
distribution over execution fragments as follows. The basic measurable events are the cones of 
finite execution fragments, where the cone of a finite execution fragment a, denoted by C a , is the 
set {a' G frags* (A) \ a ^ a'}. The probability [i a )S of a cone C a is defined recursively as follows: 



Standard measure theoretical arguments ensure that /v.s extends uniquely to the cr-field generated 
by cones. We call the measure /z^ a probabilistic execution fragment of A and we say that it 
is generated by a from s. Given a finite execution fragment a, we define ^ CT S (a) as n a:S (a) = 
A*ct,s(C q ) ■ a(a)(A-), where cr(a)(_!_) is the probability of chosing no transitions, i.e., of terminating 
the computation after a has occurred. 

We say that there is a weak combined transition from s G S to /i G Disc(S) labeled by a G 17 
that is induced by a, denoted by s ==>c M> if there exists a scheduler <r such that the following 
holds for the induced probabilistic execution fragment /i a ,s' (1) Ha,s(frags* (A)) = 1; (2) for each 
a G frags*(A), if /x CTlS (a) > then trace (a) = trace(a); (3) for each state f, /i CTiS ({a £frags*(A) \ 
last (a) = t}) = fi(t). See [22| for more details on weak combined transitions. 
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Fig. 1. The probabilistic automaton £ 



<r(a) — S± for each other finite execution fragment a. For instance, state ■ is reached with proba- 
bility p a '.s({ct G frags* {£) \ last(a) — ■}) = p a i ,g({sruaM, sTtrsruaM}) = l + 
|-1-1-|-1-1-1=^ = n(M), as required. 

We say that there is a hyper-transition from p G Disc(5) to p G Disc(S) labeled by a G S, 
denoted by p ==>c Ai, if there exists a family of weak combined transitions {s ==^c MsKeSuppfp) 
such that fx = E seS upp(p) p( s ) ' Ms- i-e., for each f G S, p(t) = E seS upp(p) p( s ) ' 

Definition 1. Let A\, Ai be two probabilistic automata. An equivalence relation TZ on the disjoint 
union S\ W S2 is a weak probabilistic bisimulation if, for each pair of states s, t € £1 W £2 such that 
s 1Z t, if s — > p s for some probability distribution p s , then there exists a probability distribution 
pit such that t ==>c At* and p s C(JZ) pt- 

Two probabilistic automata Ai and A2 are weakly probabilistic bisimilar if there exists a weak 
probabilistic bisimulation TZ on Si W1S2 such that s\ 1Z S2. We denote the coarsest weak probabilistic 
bisimulation by w, and call it weak probabilistic bisimilarity. 

This is the central definition around which the paper revolves. Weak probabilistic bisimilarity is 
an equivalence relation preserved by standard process algebraic composition operators on PA \ 19 1. 
The definition of bisimulation can be reformulated as follows, by simple manipulation of quantifiers: 

Definition 2. Given two PAs Ai, Ai, an equivalence relation 1Z on Si l±J S2 is a weak probabilistic 
bisimulation if for each transition (s, a, p s ) G D\ l+l D2 and each state t such that slZt, there exists 
p t such that t ==>c Mt an d fJ-s £{Tt) fit- 

3 Weak Transition Construction as a Linear Programming Problem 

We now discuss key elements of a decision algorithm for weak probabilistic bisimilarity. As we will 
see, the core ingredient - and the source of the exponential complexity of the decision algorithm 
of - is the recurring need to verify the step condition, that is, given a challenging transition 
s — % p and (s, t) £lZ, to check whether there exists a weak combined transition t =^c A*t sucn 
that p C(JZ) p t - 

With some inspiration from network flow problems, we will be able to see a transition t ==>c 
of the PA A as a flow where the initial probability mass St flows and splits along internal transitions 
(and exactly one transition with label a for each stream when a ^ r) accordingly to the transition 
target distributions and the resolution of the nondeterminism performed by the scheduler. 

This will allow us to arrive at a polynomial time algorithm to verify or refute the existence of a 
weak combined transition t ==>c Mt such that p £(1Z) pt- This is the core ingredient of an efficient 
algorithm for deciding weak probabilistic bisimilarity, stated in Section|U 

3.1 Allowed Transitions 

For the construction we are going to develop, we consider a more general case where we parametrize 
the scheduler so as to choose only specific, allowed, transitions when resolving the nondeterministic 
choices in a weak combined transition. This generalization will later be exploited by enabling us to 
generate tailored and thereby smaller LP-problems. 

For the intuition of this generalization, consider, for example, an automaton C that models a 
communication channel: it receives the information to transmit from the sender through an exter- 
nal action, then it performs an internal transition to represent the sending of the message on the 
communication channel, and finally it sends the transmitted information to the receiver. The com- 
munication channel is chosen nondeterministically between a reliable channel and an acknowledged 
lossy channel. If we want to check whether C always ensures the correct transmission of the received 
information, we can restrict the scheduler to choose only the lossy channel, i.e., we allow only the 
transitions relative to the lossy channel; if we impose this restriction and C is able to send eventually 



the transmitted information to the receiver with probability 1, then we can say that C always ensures 
the correct transmission of the received information. 

Definition 3 (Allowed weak combined transition). Given a PA A and a set of allowed transitions 
A C D, we say that there is an allowed weak combined transition/rom stop with label a respecting 
A, denoted by s =4>c A*> if there exists a scheduler a that induces s ==>c P such that for each 
a £ frags* (A), Supp(cr(a)) C A. 

It is immediate to see that, when we consider every transition as allowed, i.e., A = D, the allowed 
weak combined transition s ==>c A 4 * s J ust tne usua l weak combined transition s ===>c P- 

Proposition 1. Given a PA A a state s, and action a, and a probability distribution p £ Disc(S'), 
there exists a scheduler ao for A that induces s ==^c M if and only if there exists a scheduler a for 
A that induces s ==>c p- 

Similarly, we say that there is an allowed hyper-transition from a distribution over states p to a 
distribution over states p labeled by a respecting A, denoted by p ==^q p, if there exists a family of 
allowed weak combined transitions {s ==>q A t «}seSupp(p) sucn A* = S P( s ) ' A*s- 

An equivalent definition of allowed hyper-transition p ==^>q p is the following: given a PA A, 
we say that there is an allowed hyper-transition from a distribution over states p to a distribution over 
states p labeled by a respecting A if there exists an allowed weak combined transition h =^Q h p 
for the PA A h = (S U {h}, s,E,DU {h -A* p}) where h£S and A h = A U {h -A* p}. 

Proposition 2. Given a PA .4, h S, a £ X?, A C D, a«c/ p, p e Disc(S), Zef A/j fee f/ie PA 
A ft = (S U {/i}, s,£,DU{/i^ p}) ant/ A h fee A U {h A* p}. 
p =>C p exi'sfs in A if and only if h =^q h p exists in Ah- 

Example 1 (cont.). If we consider again the automaton £ in Figure Q] and the set of allowed transi- 
tions A = D\{t — 5g}, it is immediate to see that the weak combined transition s => c P where 
p = { Yg A, jqM, is not an allowed weak combined transition respecting A and that the only 

allowed weak combined transition with label a enabled by s is s p having p = { j A, jM, 

as target distribution. 

3.2 A Linear Programming Problem 

We now assume we are given the PA A, the set of allowed transitions A C D, the state t, the action a, 
the probability distribution p, and the equivalence relation 1Z on S. We intend to verify or refute the 
existence of a weak combined transition t ==>c pt of A satisfying p C(1Z) pt via the construction 
of a flow through the network graph G(t, a, p, A, TV) = (V, E) defined as follows: 

Definition 4. Given the PA A, the set of allowed transitions A C D, the state t, the action a, 
the probability distribution p, and the equivalence relation 1Z on S, we define the network graph 
G(t, a, p, A, TV) — (V, E) relative to t =^q pt of A as follows: for a ^ r, the set of vertices is 

v = {a, t}usu S tr U S a U Sf U (S/ft) 

vr/zere 

S* r = {v tr \ tr = v p £ A,b £ {a, r}}, 
«5 a = {v a v £ S}, and 

s? = {vt I v tr e s 4r } 

ant/ fne se? of arcs is 

E = {(A,t)} U {(« 0) C), (C, T) | C e « £ C} 

U{(«, u tr ), «'), (« a , «* r ), <) I = « ^> P G A,«' e Su PP (p)} 
U{(v,vT),(vT,v' a ) | ir = U ^peA, U 'eSupp(p)}. 



For a — t the definition is similar: V = {A, T} U S U S tr U (S/K) and E = {(A,*)} U 
{(«, C), (C, T) | C G S/ft, w G C] U {(«, w* r ), (v tr , vf) \tr = v -A* p e A,v' G Supp(p)}. 

A and T are two vertices that represent the source and the sink of the network, respectively. 
The graph encodes possible sequences of internal transitions, keeping track of which transition has 
happened by means of the vertices superscripted with tr, for this the set S tr contains vertices that 
model the transitions of the automaton. The subsets of vertices subscripted by a are used to record 
that action a has happened already. Notably, not every vertex is used for defining arcs: the vertices 

v tr where tr = v — —> p G A and b = a ^ r are used only to define the corresponding vertices u* r 
that are actually involved in the definition of the set E of arcs. We could have removed these vertices 
from S tr but this reduces the readability of the definition of S* r without giving us a valuable effect 
on the computational complexity of the proposed solution. 



Example 1 (cont). Consider the automaton £ in Figure Q] and suppose that we want to check 
whether there exists an allowed weak combined transition s ==^c p such that p £(1Z) p where 
p = { Yg A, jqM, y§^} and the classes induced by 1Z are {{s, t, u, v}, {A}, {■}, {^}}- Let tro = 
s -A» {it, ju, hv}, tri = t -A> S A , tr2 = u -A> fi u , tr% = v — 5+, and tr± = t -A> $ s . The 
network G(s, a, p, D, TV) is as follows, where we omit vertices A, ■, and # since they are not 
involved in any arc. Numbers attached to arcs indicate probabilities, and are not part of the graph. 




Our intention is to use the network G(t, a, p, A, TV), in a maximum flow problem, since solv- 
ing the latter has polynomial complexity. Unfortunately, the resulting problem does not model an 
allowed weak combined transition because probabilities are as such not necessarily respected: In or- 
dinary flow problems we can not enforce a proportional balancing between the flows out of a given 
vertex. Instead, the entire incoming flow might be sent over a single outgoing arc, provided that the 
arc capacity is respected, while zero flow is sent over other arcs. In particular, we have no way to 
force the flow to split proportionally to the target probability distribution of a transition when the 
flow is less than 1. Apart from that, there is no obvious way to assign arc capacities since imposing 
capacity 1 to arcs is not always correct even if this is the maximum value for a probability. This prob- 
lem is specifically caused by cycles of internal transitions. For self loops like s — > p with p(s) > 0, 
one might after some reflection come up with a capacity 1/(1 —p) where p — p(s), but this does not 
extend to arbitrary r-connected components. 

For these reasons, we have to proceed differently: Since any maximum flow problem can be ex- 
pressed as a Linear Programming (LP) problem, we follow this path, but then refine the LP problem 
further, in order to eventually define a maximization problem whose solution is indeed equivalent 
to an allowed weak combined transition, as we will show in Section [331 For this, we use the above 
transformation of the automaton into a network graph as the starting point for generating an LP prob- 
lem, which is afterwards enriched with additional constraints: We adopt the same notation of the max 
flow problem so we use f UiV to denote the "flow" through the arc from u to v. The balancing factor 
is a new concept we introduce to model a probabilistic choice and to ensure a balancing between 
flows that leave a vertex representing a probabilistic choice, i.e., leaving a vertex v G S tr U S%~. 



Definition 5 (The t <s> C{K) p LP problem). For a ^ t we define the t A* LP 

problem associated to the network graph (V, E) — G(t, a, p, A, 1Z) as follows: 



m&x J2(x,y)eE ~fx,y 

under constraints 

f U v>0 for each (u, v) G E 
h,t = 1 

fc,j — p{C) foreachC G S/1Z 

T<ue{x\{x,v)eE} U,v ~ Eu6{»|(t,,»)eB} /«.« = /oreac/i »£F \ {A, T} 

fv tr ,v' — p( v ') • /u v tr — for each tr = v — > p G A andv' G Supp(p) 

f v try — p{v') ■ fv a .v tr = /or eac/j tr = v — p G A ant/ i/ G Supp(p) 

f v tr v i — p(v') ■ f v . v tr — for each tr = v — ^ p G A and v' G Supp(p) 



The constraints as Y, u e{x\(x,v)eE} ~ E Me { s |(«, 9 )g£} = for v G V \ {A, ▼} are 
also known as conservation of the flow constraints. When a is t, the LP problem t =>c ^(^-) A* 
associated to G(t, r, /z, A, 7?.) is defined as above without the last two groups of constraints. Note 
that the constraints of t ==>c ^(^) P define a system of linear equations extended with the non- 
negativity of variables f UjV and this rules out solutions where some variable f XjV has an infinite 
value. Moreover this may be used to improve the actual implementation of the solver. 

We can define the objective function in several ways but this does not affect the equivalence of 
t =>c o C(TZ) p and allowed weak combined transitions: in fact, the equivalence is based on vari- 
ables fv a ,[v]n an d/c,Y (where v G S'andC G S/1Z) that represent the probability to reach each state 
v (and then stopping) and each equivalence class C, respectively; by definition of t ==>c ^(^-) A* 
we have that J2 v ec fv a ,c — fc.i an d fc.j = p{C), thus their value does not strictly depend on the 
objective function. When a = r, we have the same result, just replacing v a with v. 

The objective function we use allows us to rule out trivial self-loops: suppose that there exists a 
transition tr — x 5 X G A that we model by arcs (a;, x tr ) and (x tr , x). The balancing constraint 
for such arcs is f x tr x — 1 • f xx tr = that is satisfied for each value of f x tr , x = fx x ir \ however, 
the maximum for the objective function can be reached only when f xx tr = 0, that is, the self-loop 
is not used. Similarly, we obtain that the value of the flow involving vertices that can not be reached 
from the vertex t is null as well as when such vertices may be reached from t but the solution of the 
problem requires that the flow from the vertex t to them is null. 

It is worthwhile to point out that the objective function max^]^ «)eJS ~fx,y is actually equiva- 
lent to minj^j. y\^E fx,y> i- e -> a weak transition can also be seen as a minimum cost flow problem 
plus balancing constraints. 

Example 1 (cont.). Consider again the automaton £ in Figure[T]and suppose that we want to check 
whether there exists an allowed weak combined transition s =>c P sucn tnat P ^Oty A 4 where 
p = { yg A, jgM, y§^} and the classes induced by TZ are {{s, t, u, v}, {A}, {■}, {^}}- Let tr Q = 

s —> {jt, ju, ^v}, tri = t -A> S A , tr2 = u — ^> 8 m , tr% = v — > <5#, and tr± = t S s - 
Besides other constraints, the LP problem s =>c ® jC(72.) p has the following constraints: 
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A solution that maximizes the objective function sets all variables to value except for 
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The variable f s ,s tr o = 20/16 is part of a cycle and its value is greater than 1, confirming that 1, the 
maximum probability, in general is not a proper value for arc capacities. 

3.3 Complexity of the LP Problem 

We analyze the complexity of the t ==>c * £(72.) p LP problem when a ^ t since t =>q o A 4 
is just a special case of t * ^■{^■) A 1 - 

Given the automaton A and the set A C 7J of allowed transitions, let As = Na = \A\, 
and A = max{As, A^}. Suppose that a ^ r and consider the network graph G(A, a, p, A, 7?.) = 
(V, E). The cardinality of V is: | V| < 2 + A s + N A + N s + N A + N s G 0(A) and the cardinality 
of E is: \E\ < 1 + 2A S + 2(A S + 1) A A + (A s + 1) N A G 0{N 2 ). Note that this is also the cost 
of generating the G(t, a, p, A, TV) network graph from the automaton A. 

Now, consider the t =>c * ^C^-) M LP problem: the number of variables is {/„,!, | (it, v) G 
E}\ = \E\ G C(N 2 ) and the number of constraints is \E\ + 1 + N s + N S N A + N S N A + N S N A + 
\V\-2e C(A 2 ), so generating t <> £(ft) p is polynomial in A. Since there exist polynomial 

algorithms for solving LP problems [25], solving the t ==>q o C(TV) p problem is polynomial in A. 

Theorem 1. Given a PA A an equivalence relation 1Z on S, an action a, a probability distribu- 
tion p G Disc(5), a set of allowed transitions A C D, and a state i 6 S, consider the problem 
t ==>q * £(7^-) A* as defined above. Let A = maxUS*!, \ A\\. 

Generating and checking the existence of a valid solution of the t * ^C^) A 4 LP problem 

is polynomial in N. 

3.4 Some Optimizations. 

The implementation of t =>c * £(7V) A* can be optimized in several ways: we can safely remove 
each constraint f UtV > when (u,v) G {{v tr ,v') tr = v — —> p G A, p(v') > 0} since it is 
implied by f v v * > and f v try — p(v')f v yr = as well as when (u,v) G {(v^,v' a ) \ tr — 
v p G A or tr = v p G A, p(v') > 0}; as second optimization, we can avoid the constraint 
fu,v > when u = C <E S/TZ and v = T since this is implied by fc t j = p(C). These optimizations 
allow us to save upto2|5|(l + |A|) constraints but the advantage we gain from them depends on the 
actual implementation of the LP solver. 

Constraints of the form £t,e{x|(x,t.)ei5} U,v ~ J2ue{y\(v, v )eE} U,u = for v G S tr can be 
removed safely since they derive from f v «- v > — p(v')f v y.r = and the fact that by construction 
there is only one arc that ends in v tr . The same holds for u* r G S%" given a ^ r, so we can skip the 
generation of up to 2\A\ constraints. 

The last optimization does not involve the removal of a constraint but only the generation of the 
LP problem itself. Given a ^ r, the subgraph whose arcs have both vertices in S a U S"* r is simply a 
copy of the subgraph whose arcs have both vertices in S U S tr , so we can speed up the LP problem 
generation by just copying a previously generated encoding. Similarly, the subgraph obtained by 
encoding internal transitions like s —> p does not depend on neither the state t, the action a, the 
probability distribution p,, nor the equivalence relation 1Z, so it can be generated only once and then 
is simply copied in the actual instance of the t ==>q * jC(72.) p LP problem. All these optimizations, 
however, do not change the complexity class of generating and then finding a feasible solution of the 
t =>c * A 4 LP problem, which remains polynomial. In any case they can improve the actual 

computation time of an implementation. 



3.5 Equivalence of LP Problems and Weak Transitions 



In this section we present the main theorem that equates t ==>q o fi with an allowed weak 

combined transition, that is, t ==>c * A* has a solution if and only if there exists a scheduler a 

for „4 that induces an allowed weak combined transition t ==>c A 4 * sucn ma t £(7£) /it. This result 
easily extends to ordinary weak combined transitions and hyper- transitions. 

Theorem 2. Given a PA A, an equivalence relation 1Z on S, an action a, a probability distribu- 
tion fi G Disc(S'), a set of allowed transitions A C D, and a state t £ S, consider the problem 
fi as defined above. 

t ==>c * A 4 nas a solution f* such that fcj— ^{C) for each C G S/1Z if and only if there 

exists a scheduler a for A that induces t ==>q pt such that p £(7V) fit- 
Proof (Proof outline). The scheduler a we define in the proof for the "only if" direction assigns 
to each execution fragment a with last (a) = v the sub-probability distribution over transitions 
defined, for each transition tr G A such that src(tr) = v, as the ratio /* v t r /f^ t , given that /* t > 0, 

where /* is the total flow incoming v, t = trace(a), and t is the concatenation of trace(a) and 
of trace(act(tr)). The remaining probability of stopping in the state v is exactly /* t u^//^. The 
way we generate the network G(t, a, p, A, TV) ensures that /* v ,, r — when t ^ {e, trace(a)} and 

that f* t [y]^/ fv t = when t ^ trace(a). The proof for the "if" direction is the dual, that is, we 
define a feasible solution /* as the sum of the probabilities of the cones of execution fragments, i.e., 

fv b = J2 a e{4>efrags'(A)\trace(^bMast(^v} AV,i(0*); then the existence of such feasible solution 
is enough to prove that there exists a (possibly different) solution f° that maximizes the objective 
function while preserving the property that for each C G S/1Z, fcf = A*(C). 

For the detailed proof, see Appendix IE1 □ 

It is worth to observe that the resulting scheduler is a determinate scheduler and an immedi- 
ate corollary of this theorem confirming and improving Propo sition 3 of |3| is that each scheduler 
inducing t ==>q pt can be replaced by a determinate scheduler inducing t ==>q fit as well. 

Example 1 (cont.). It is interesting to observe that the same weak combined transition can be gen- 
erated by different schedulers: we already know from the first part of this example that there exists a 
scheduler a inducing s =>q fi where fi = { A A, A>, 

Let again tro = s — > {4t, \u, kv}, tr\ = t — > S A , tr-2 = u —> 5 m , tr% = v — <5#, and 
Ar4 = t -A> S g . Theorem |2] ensures that there exists a scheduler a', possibly different from a, that 
induces s =>c M! m particular, a' is the determinate scheduler defined as follows: 

S tro if trace(a) = e and last (a) = s; 

{girt, 5^4} if trace(a) = e and last (a) = t; 

a (a) — \ 5 t r 2 if trace{a) = e and last (a) = u; 

Str 3 if trace(a) — e and last(a) = v; 

5± otherwise. 

It is straightforward to check that a' actually induces s ==>c P- For instance, state A is reached 
with probability p„i G frags*(£) | last(a) = A}) = fi a i g ({sTt(TSTt) n aA | n G N}) = 
1 • \ ■ E„ 6 n(| • 1 • 1 • IT ■ \ ■ 1 • 1 = I ■ \ ■ (1 - ir 1 =l-i-I = TE= M(A), as required. 

Corollary 1. Given a PA A t G S and h £ S, a G S, p, /i, fit G Disc(S'), A C D, an equivalence 
relation TZ on S, a transition h p, Ah = A U {h — p}, = D U {h — p}, and the PA 
Ah = (S U {h}, s, S, Dh), the following holds: 



1. t ==>q * £(7t) P has a solution f* such that f£ T = p(C) for each C G S/7Z if and only if 
there exists a scheduler a for A inducing t =>c l^t such that p £(1Z) pt; 

2. h ==>Q h * £{TVj A* (h = =^c h * ^(^) I 1 ) relative to Ah has a solution f* such that f£ v = 
p(C) for each C G S/1Z if and only if there exists a scheduler a for A inducing p pt 
(p ==>c A't> respectively) such that p t C(TZ) p. 

When 1Z is the identity relation I, p C(I) pt implies p t = p. 

Proof (Proof outline). The corollary follows directly from a combination of Theorem|2]for the equiv- 
alence between the LP problem and allowed weak combined transition, PropositionQ]for weak com- 
bined transitions, and Proposition[2]for hyper-transitions. □ 

3.6 Equivalence Matching 

Theorem [2] and its corollary allow us to check in polynomial time whether it is possible to reach 
a given probability distribution p from a state t or a probability distribution p. We now consider a 
more general case where, given a PA A, two distributions p\ , p2 G Disc(S'), two actions a\ , a-i G S, 
two sets A\ : A2 G D of allowed transitions, and an equivalence relation 1Z on S, we want to check 
in polynomial time whether there exist pi, P2 G Disc(S') such that p\ pi, P2 ==^c 2 A*2> and 

pi C(TZ) p2- In order to find pi and p2, we can consider a family {pc}ces/iz °f non-negative 
values such that 2~2ces/nPc — 1 and a probability distribution p satisfying p(C) — pc for each 
C G S/K and then solve p x * C ( n ) P and P2 =^c 2 * C ( n ) A wh ere p <s> £(K) p 

is the problem h o C{U) p relative to A h = {S U {h}, s,S,D U {h p}) with h £ S 

and Ah = AU {h p}. The main problem of this approach is to find a good family of values 
PC', since we do not care about actual values, we consider pc as variables satisfying pc > and 
Z~2ces/izPc = 1 a nd we define the LP problem P\ 2 derived from P\ = p\ ==^ c 1 o P- and 

p as follows (after renaming of P2 variables to avoid collisions): the objec- 
tive function of Pi 2 is the sum of the objective functions of Pi and P2, the set of constraints of Pi. 2 
i s 2~2ces/iz Pc = 1 together with pc > for C G S/1Z and the union of the sets of constraints of Pi 
and P2 where each occurrence of /2(C) is replaced by pe- 
lt is quite easy to verify that P\p has a solution if and only if both Pi and P2 have a solution 
(with respect to the same p) and thus, by Corollary [TJD, if and only if pi and P2 enable an allowed 
hyper-transition to pi and P2, respectively, such that pi C(TZ) p2, as required. It is immediate to see 
that Pi 2 can still be solved in polynomial time, since it is just the union of Pi and P2 extended with 
at most N variables and 2N constraints where N = \S\. 

Proposition 3. Given a PA A two distributions pi, P2 G Disc(S*), two actions 01, a% G S, two sets 
Ai, A2 G D of allowed transitions, and an equivalence relation 1Z on S, the existence of pi, P2 G 
Disc(5) such that pi =>q 1 pi, P2 =>c 2 an d Mi ^2 can be checked in polynomial time. 

The above proposition easily extends, by Corollary Q] to each combination of weak combined 
transitions, allowed hyper-transitions, and allowed weak combined transitions as well as to exact 
matching as induced by the identity relation I. 

4 Decision Procedure 



In this section, we recast the decision procedure of [3 1 that de- 
cides whether two probabilistic automata A% and A2 are bisimilar 
according to rj, that is, whether Ai ~ A2, following the standard 
partition refinement approach [3. 16. 18. 20|. More precisely, pro- 
cedure Quotient iteratively constructs the set S/~, the set of 
equivalence classes of states S = S\ tfcl S2 under «, starting with 



QUOTIENT(^4i,^t 2 ) 

W= {S1WS2}; 

(C,a,p) = FindSplit(W); 

while C / do 

W= REFINE(W, (C,a,p)); 

(C,a,p) = FindSplit(W); 
return W 



FindSplit(W) 



the partitioning W= {S} and refining it until W satisfies the definition of weak probabilistic bisim- 
ulation and thus the resulting partitioning is the coarsest one, i.e., we compute the weak probabilistic 
bisimilarity. 

Deciding whether two automata are bisimilar then 
reduces to checking whether their start states belong to 
the same equivalence class. In the following, we treat 
W both as a set of partitions and as an equivalence 
relation without further mentioning. 

The partitioning is refined by procedure REFINE 
into a finer partitioning as long as there is a parti- 
tion containing two states that violate the bisimulation condition, which is checked for in proce- 
dure FindS plit. Procedure Refine, that we do not provide explicitly as in 0, splits partition C 
into two new partitions according to the discriminating information (C,a,/i) identified by FlND- 
Split before. So far, the procedure is as the DecideBisim(Ai , A 2 ) procedure proposed in 0. 

The difference arises inside the procedure FindSplit, where we check directly the step condi- 
tion by solving for each transition s fj, the LP problem t =^>q o C(W) /1 that has a solution, 
according to Corollary [TJT}, if and only if there exists t ==^c A*t sucri that /i C(W) /if. 



for all (s, a, ^) G D = Di U D 2 do 
for all t e [s]w do 

if £ ==>q «> C(W) fi has no solution 
return ([s]w,a,fi) 
return (0, r, 5s) 



4.1 Complexity Analysis of the Procedure 

Given two PAs Ai and A 2 , let S = Si W S 2 , D = D x W D 2 , and N = m&x{\S\, \D\}. 

In the worst case (that occurs when the current W satisfies the step condition), the for at line 1 of 
procedure FindSplit is performed at most N times as well as the inner for, so t ==>c * £(VV) /i 
is generated and solved at most N 2 times. Since by Theorem Q] generating and checking the ex- 
istence of a valid solution for t «> C(W) /j, is polynomial in N, this implies that also Find- 
Split is polynomial in A^; more precisely, denoted by p(N) the complexity of t * jC(W) /1, 
FindSplit g 0(N 2 p(N)). Note that we can improve the running time required to solve the 
t ==>c * £(W) LP problem by replacing D with D' at line 3 of FindSplit where D' contains 
only transitions with label r or a enabled by states reachable from t. 

The while loop in the procedure QUOTIENT can be performed at most N times; this happens 
when in each loop the procedure FindSplit returns (C, a, /1) where C ^ 0, that is, not every pair of 
states in C satisfies the step condition. Since in each loop the procedure REFINE splits such class C 
in two classes C\ and C 2 , after at most N loops every class contains a single state and the procedure 
FindSplit returns (0, r, 5 S ) since each transition s /.t s is obviously matched by s itself. Since 
Refine and FindSplit are polynomial in N, also Quotient is polynomial in N, thus checking 
A\ « A 2 is polynomial in N . 

Theorem 3. Given two PAs A\ and A 2 , let N = ma^{\Si W S 2 \, \ D\ W D 2 \}. 
Checking Ai ~ A 2 is polynomial in N. 



5 Concluding Remarks 

This paper has established a polynomial time decision algorithm for PA weak probabilistic bisimula- 
tion, closing the quest for an effective decision algorithm coined in [3 1. The core innovation is a novel 
characterization of weak combined transitions as an LP problem, enabling us to check the existence 
of a weak combined transition in polynomial time. The algorithm can be exploited in an effective 
compositional minimization strategy for PA (or MDP) and potentially also for Markov automata. 
Furthermore, the LP approach we developed is readily extensible to related problems requiring to 
find a specific weak transition. Another area of immediate applicability concerns cost-related prob- 
lems where transition costs may relate to power or resource consumption in PA or MDP. 
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A Equivalences between Allowed Transitions and Ordinary Transitions 



Result 1 (Proposition© Given a PA A, h £ S, a £ S, A C D, and p, p £ Disc(S'), let Ah be the 
PA A h = (SU {h}, s,i;,DU{/iA p}) and A h be A U {h -A p). 
p =>q p. exists in A if and only if h ==^-Q h p exists in Ah- 

Proof. A common result we need is that for a = sqciiSi . . . such that first(a) = so £ Supp(p), 
a £ frags* (A) if and only if lira £ frags* (Ah)', denote by s_i the state h and by ao the action r 
so hra is just S-iaoSoaiSi .... Since a £ frags* (A) we have that for each < i < \a\ there exists 
(sj, Oj+i, /ij+i) such that pi + i(si + i) > 0. Since p(sq) > 0, then for each — 1 < i < \a\ there exists 
a transition (sj, aj+i, /Ltj+i) such that pi + i(si + i) > 0, so hra £ frags* (Ah). 

Now, suppose that frra £ frags* (Ah). This implies that — 1 < i < \a\ there exists a transition 
(sj, Oj+i, fXi+i) such that /ij_)_i(si+x) > 0; in particular, it holds that < i < \a\ there exists 
a transition (sj, di+i, Pi+t) such that pi + i(si + i) > and this implies that so £ Supp(p) and 
a £ frags* (A). 

It is straightforward to check that given an automaton B, a scheduler a, and a state s, for each 
a £ frags* (B), p a ,s(Ca) > implies first(a) = s that is implied by p ayS (a) > as well. 

(=>) By definition of p p there exists a family {s ==>c MsKeSuppfp) °f allowed weak tran- 
sitions such that p = X)seSupp(p) p( s )Ps- This implies that there exists a family of sched- 
ulers {°s} sE s U pp(p) such that for each s £ Supp(p), a s induces the allowed weak transition 

a A 

s — fQ p s . 

Let a be the scheduler for Ah defined as follows: 

{5, t if a = h, 
h — >p 
<J s (a!) if a = hra' = }iTsa\S\ . . ., 
5j_ otherwise. 

To prove that a actually induces the allowed weak transition h =>Q h p, we need of some 
preliminary result: for each finite execution fragment a £ frags* (Ah), Supp(cr(a)) C Ah- In 
fact, Supp(er(/i)) = {h p) C A h ; Supp(a(hra')) = Supp(cr s (a')) Q A C A h where 
s = first(a'); for all other execution fragments, Supp(<r(a)) = Supp(<5x) = Q Ah- 
Another result we need is the following: for each a £ frags* (A), if first(a) — s, then 
Ha,h(ChTa) = p(s)Pa s ,s(C a )- We prove this result by induction on the length n of a: if n = 0, 

then /J, a ,h( C hrs) = H*M( C h)Y.treD{T) (J ( h )( tr ) ' Ptr(s) = ^Y,treD(r) <T ( h )( tr ) ' ^tr(s) = 

p(s) = p(s)p a ^ s (C s ); if n > 0, then there exists a' such that a — a 1 at for some action a 

and State t, SO pa,h(Chra) = Pcr.h(Chra'at) = Vo,h{ChTa')Y,tr£D{a) &(hTa')(tr) ■ p tr (t) = 
P(s)fJ-<7 s ,s(C a <)J2treD(a) Os(u')(tr) ■ p tr (t) = p(s)p <J ^ s (C a ' a t) = p(s)p a ^ s (C a ). 

Now we are ready to show that the three conditions on the probabilistic execution fragment p a ,h 
induced by a are satisfied. 
1. 

Pa,h(f ra 9 s *(^h)) 

= ^ PaJi(C a ) ■ <7(a)(lf) 
a£fmgs* (Ah) 

= Hcr,h(Ch) ■ 0-(h)(±) + ^ Pa,h( C hra) ■ Cr(hTa)(±) 

hra£frags* (Ah) 

= + p(first(a))p IJfir3t(a ^ first ( a) (C a ) ■ a(hra)(L) 

hraZLfrags* (Ah) 

p(first(a))p afirBt{a)tfirst{a) (C a ) -a(hTa)(±) 

hra^frags* (Ah) 



= E E p{s)Pa s A c A ■ cr(hTa)(±) 

sES a£{a' efrags* (A)\first(a')=s} 
s£S a£{a' S/rags* (A)\first(a')=s} 

= J2p( s ) E ^,A a ) 

seS a£{a' efrags* (A)\first(a')=s} 

= J2p( s ) E »*.,>(<*) 

seS a£frags*(A) 

ses 

= E^) 1 

ses 
= 1; 

2. let a' £ frags*(Ah) such that p, a A a ') > 0' this implies that first(a') = h thus a' = /ira 
for some a E frags* (A) since h p is the only transition enabled by h. (J, a ,h( a ') > 
implies as well that first{a) = s£ Supp(p) and n <TeiS (a) > for some state s hence, by 
definition of ,s ==>c Ms, trace(a) = trace(a) = trace(a'), as required; 

3. 

M<r,/i({a G frags*(A h ) \ last (a) = q}) 
= E H*Ji(C a ) ■ o-(a)(-L) 

{a G/rags * (Ah ) I Zast ( a ) =q } 

= E ^/.(C/ira) • a-(/ira)(±) 

{/iraG/rags*(-4/ 1 )|/ast(a) — g} 

E PC/ Jrs *( Q! ))Ma /irat(oi) ,/ir S i(a)(C a )-0-(/lTa)(±) 
{/iraG frags* {Ah)\lo,st(a)=q} 

E p(first(a))p 

& first (a) , first (a) 

(C a ) ■ a(hra)(±) 

{hrce^frags* (Ah)\last(ot)=q} 

= E E p(s)/"a s , s (C a ) -a(/ira)(±) 

s£S a£{a'£/rags*(.4) |/ii-st (a')=sA(asi(a')=g} 

= E^ S ) E /^.,.(Ca) •*(<*)(-!-) 

s£S ae{a'e/rags*(.4)|/irst(a')=sAiast(c(')=<?} 

= E^ s ) E v<TsA a ) 

s£S a£{a'S/rags* (.A) \first(a')=s/\last(a')=q} 

= E^ S ) E ( a ) 

s£S aE{a' efrags* (A)\last(a')=q] 

= J2p( s )Ps(q) 

ses 

= M<?)- 

(<*=) For each s 6 Supp(p), let cr s be the scheduler for A defined as follows: 



a(hra) if first (a) = s, 
5j_ otherwise. 



a s (a) = 
scheduler; 

need of some preliminary result: for each execution fragment a e frags* (A), Supp(a(a)) C 



To prove that the family of schedulers a s induces the allowed hyper transition p ==>c M> we 



A. In fact, Supp((T s (a)) = Supp (er(/iTa)) C Ah where s — first(a); by hypothesis, 
ft S and this implies that for each s p s G £>, ft, ^ Supp(/x s ), hence ft, — p ^ 
SubDisc(£)(Zasi(o!))), so ft — p £ Supp(cr(ft,Ta)) and thus Supp(a s (a)) C A. For all other 
execution fragments, Supp(<r(a)) = Supp(<5^) = C A. 

Another result we need is the following: for each a G frags* (A), if first (a) — s, then 

prove this result by induction on the length n of a: if n = 0, 

P\ s ) 

then At«r,h(Cftr«) _ M<r,h(Cfe) Etre j(r) ffWW ' Ms) _ lStre/?(r) g ( ft )(^) -Mtr(a) _ 
p(s) p(s) p(s) 

= I = [la- S (C S ); if n > 0, then we have that a = a 1 at for some action a and state 

P{ s ) 

i, therefore = — — = ^ = 

p(s) p{s) p(s) 

pa s ,s(C a 'at) = Pcr s ,s(C a )- 

Now we are ready to show that the three conditions on the probabilistic execution fragment p a ^ s 
induced by a s are satisfied, where p s is defined for each t G S, as follows: 

Pa h({hTa' G frags* (Ah) \ last (a 1 ) = t A first (a') = s}) 



P«s,s(frags*(A)) 

= E Pa e ,s(C a ) ■a s (a)(±) 

a£{a' £ frags* (A) \first(a / )—s} 

' ■a(H(l) 

/i r a G { ft- r a ' G /rap s * ( .A ft. ) | fi rs t ( a ' ) — s } 



E 



PaJi(Chra) ■ a(hTa)(±) 



hraE{hTa , Efrags*(A-h)\first(a')=s} ^ ^ 

_ J2hTa£{hTa>£frags*(A h )\first(a') = s} P<?,h{ChTu) ' Cr(ft.TO:) (_L) 

P(*) 

P(f) 

= 1; 

2. let a G frags* (A) such that p a ^ s (a) > 0; this implies that first(a) = s and paj^hra) > 
0, hence trace(a) = trace(hra) = trace(a), as required; 

3. 

Pa s ,s{{ot G frags* (A) \ last (a) = q A first (a) = s}) 

E P<?sA c «) ■ o- s (a)(-L) 

{aG/raps* (.4) | ?asi(a)— q A first (a)— s} 

• -a(H(l) 

{hra£ frags* (Ah)\^o>st(a.)—qAfirst(a.)—s} 

M<T,h(CW a ) • o-(ftra)(±) 



E 



{hra£ frags* (Ah)\lo>st(a)=qAfirst(a)=s} ^ ^ 



_ J2{hTa£frags'(A h )\last(a)=q/\first(a)=s} Pcr,h(Ch T a) " a(hTa)(±) 

Pis) 

^a,h({hra £ frags* (Ah) \ last(a) — q Afirst(a) = s}) 

The final step is to prove that fj, — X)sGSupp(p) p( s )Ms. that is, for each state t £ S, it holds that 

/*(*) = E. e supp( P )/>(«K(*): 

sGSupp(p) 

= 5^ /9(s)/z - SiS ({a G frags*(A) | last(a) = t/\ first(a) = s}) 

sGSupp(p) 

sGSupp(p) a£E{a' £ frags* (A.) \ last (a')—t/\ first (a' ')— s} 

- £ " <s) s 

s^Supp(p) a£{/iTQ' £ frags* (A) \ last(a')—tAfirst(a')—s} 

sGSupp(p) q£{/itq' £ frags* (A.) \ last(a.')—t/\first(a')—s} 

= P<T,h( a ) 

a£{hra' £frags* (A) \ Last(a f )—t] 

= pa,h({a € frags* (A) | last (a) = t}) 
= P(t), 



as required. 



□ 



Result 2 (Proposition [T) Given a PA A a state s, and a probability distribution fx 6 Disc(S'), there 

D 
c 

f/zaf induces s =>c M- 



exisfs a scheduler arj for A that induces s ==^c ^ if and only if there exists a scheduler a for A 



Proof. The fact that the existence of s =>c P implies that there is s ==>c A* is immediate, since 
by definition of allowed transition, s ==>q /i requires the existence of a scheduler a that induces 

a 

s =^ c A*- 

For the other implication, it is enough to verify that er satisfies the condition: for each a £ 
frags* (A), Supp(<7(a)) C D. This is obviously true since by definition of scheduler a(a) £ 
SubDisc(D) holds, so Supp(cr(a)) CD. □ 

B Proof of Results Enunciated in Section |3] 

Result 3 (Theorem© Gi ven a PA A an equivalence relation 1Z on S, an action a, a probability 
distribution fi £ Disc(S'), a set of allowed transitions A C D, and a state t £ S, consider the 
problem t =>q «• £(Tt) A* as defined in Section\3\ 

t =^q <=> jC(TZ) /i has a solution f* such that f£ T = ji(C) for each C £ S/1Z if and only if there 
exists a scheduler c for A that induces t ==>q pt such that fi C(TZ) [it- 
Proof Given a solution /* of t ==>c ® C(TZ) p, denote by /* the value /* = J2 u ev i- e -> me 
total incoming flow in the node v. 



(<S=) Let a be the scheduler that induces the weak transition t => c Pt and be the probabilis- 



tic execution fragment generated by a from t. For each finite execution fragment 
M<r,t(C^) > 0, denote by <f> the last state last((j>) of </> and define as follows: 



such that 



ft = I 

J x,y 



(1 ifx 

M<T,t(C^)c7(0)(-L) ifx 

M<7,t(C^)CT(0)(-L) ifx 

Pa,t( C 4>) <T ( < l ) )( tr ) ifx 

Vo,t{C<i>)a{4>){tr)p{q) ifx 

^<y,t{C c j > )a(4>)(tr) ifx 

lJ-a,t{C<t>)cr{(t>){tr)p(q) ifx 

P<rA C 4>) (T ( < t ) )( tr ) ifx 

lJ-a,t{C (t> )a{(j)){tr)p{q) ifx 

k otherwise. 

Finally, define as 



=A, y = = t; 

4>,y = [4>}k, and a = r; 
--4> a ,y= [4>]n, and a ^ r; 
- 4>,y — (f> tr , trace((j)) — e, and tr = 



</, trace(<p) = e, and tr = <fi 
— <^>* r , trace{4>) = e,tr = <f> p, and a ^ r; 
2/ = 9o) trace(<f>) = e, tr = <j) p, and a ^ r; 
a ,y = 4>a '> trace{4>) = a ^ r, and tr = 4> — p; 
a, y = q a , trace(<t>) = a / r, and tr = <f> — ^> p; 



0, y 



^(pEfrags* (A) fx,y 



ifx = Ce S/TZ andy 
otherwise 



It is straightforward to verify that the definition of f X:V given above implies that f X:V > for 
each (x, y) e that / A;t = 1, and that /c, T = for each C e S/7£. 

Now consider the constraint f v *ry = p(v')f v _ v t r for tr = v p e D and u' e Supp(p). 
There are two cases depending on whether an execution fragment <f> satisfies v = last ((f)) 
and p, a ^{C^) > 0. If <fi satisfies v = last(<f>) and /v.t(CV) > 0, then by definition we have 
ftyr = j u CT ,t(C^)<T(^)(ir) and f* ry = p a , t (C<i,)a(4>)(tr)p(v'), thus f* ry = p{v')f* vtr , 
as required. If <fi does not satisfy the conditions, then f£ vtr — and f^ tr v , = 0, hence 
again fj/ tT , = p(v')f^ vtr . This implies, together with the definition of f XtV , that = 

T,4,efrags*(A) f^,v> = T.^frags* (a) PW)ft,v» = pWfv.v", as required. The cases = 
p{v' a )fv a ,v tT an ^ /u*>' = p{ v ')fv,v tr are similar. 

The remaining part of this proof considers the so called conservation of the flow constraints, i.e., 

constraints of the kind E u6 {x|(x,»)6B} /«,* = I^ u e{ s/ |(«,y)e£;} /«,« for each " e F \ {A, ▼}. 
There are several cases (comments refer to the previous equality): 
case v = C e S/TZ: 



E 

u£{y\(C,y)eE} 



by definition of E 



by constraint on fc.y 



by definition of p t and of p a ,t(<f>) 



fc,u — JC,1 



53 ^, t (CVM0)(-L) 

{0G/rags*(^t)|;ost(0)eC} 



V ft + ft 

{<t>e frags* (A)\<PeC} 



by definition of f£ 



by definition of E 



E E ft,C + ft a ,C 

zeC {4,efrags*(A)\$=z} 

e e /«,c 

u£{a:|(i,C}e£} {4>£frags* (A)\4>=u} 



e e /«,c 

ti£{a;|(2;,C)GE} <^G/rags*(.4) 



by definition of 



case v = x tr for tr = x — ^ p: 



E ^ u - c 

u£{x\(x,C)£E} 



^ ^ fu,x tr /a;, a 

u£{z|(z, a "')££} 



by definition of E 



by definition of f x , y 



= y p * 

cj>£frags* (A) 



V f 4, 

/ j ■> x,x tr 

{4>€frags* (A) | last(<f>)—x} 



since f£ = when last ((f)) ^ x 



E ^,t(C )(7(0)(*r) 

{4>£frags* (A) \ last(<f>)=x} 



by definition of 



{4>Efrags* (A) \ last(4>)=x} a^'GSupp(p) 



since £x'€Sup P („) POO = 1 



E E ^(^M^M*') 

{</>£ frags* (A) \ last(4>)=x} x' GSupp(p) 

= E E liaAC+MMtrMx') 

x' GSupp(p) {<p£frags* (A) \ last(ip)—x} 

~ fx tr ,x' 
x' GSupp(p) {<fi£frags* (A) \ last(4>)=x} 



by definition of ft r , 



a:'eSupp(p) <f>£frags* (A) 



since ft r , = when last ((f)) ^ x 



by definition of f x *r x i 



£c'GSupp(p) 



ie{z|(x tr ,z)e.E} 



by definition of _E 



case v ~ x^: the proof is analogous; 
case v = t: 

e = E X! 

«e{y|(t,2/)e-E} tte{y|(t,y)G.E}0e/rags*(.4) 



by definition of f t ,u 



0e/ra S s*(^) ue{y\(t,y)eE} 

= E I + E ^t" 

<pefrags* (A) \ {t tr \tr=t^pjast(<j,)=t} 



by definition of by definition of /T tr , 



XI 4[t] TC + E E £ 

0G/rags*(.4) <p£frags*(A) {ttr\ tr =t^ P Jast(<p)=t} 

E /*„, t (C*M0)(-L) 

{0G/rags*(^l)|iost(0)=t} 

+ E E naAc^mtr) 

4>efrags' (A) {tr=t^ p\last(<f>)=t} 



by definition of //[ t j w and of / t ^ (r 



{0G/rags*(^l)|iast(0)=t} \ {tr=t^p} 

E V<T,t( C <t>) 
{4>£frags* (A) \ last(<t>)=t} 



since a(0)(_L) = *W(*r) 



=/v,t(Ct) + E /v,*(CV) 

{0e/rags*(.4)|0=0'-ri} 

=1+ E ^,t(C O E <r(4>')(tr)p(t) 

{4>efrags" (A)\4>=<p' rt} {tr=x^p\x=last(<f>')} 



by definition of \i a ,tiC<^) 



=h,t+ E E ii a , t {Cv)a{<lf){tr)ptt) 

{4>£frags*(A)\<i>=4>'Tt} {tr=x^p\x=last(<p')} 



by definition of / Aj 



=/A,i + E E ^* tr ,« 

{0e/ra 9 s*(^)|0=0'Tt} {tr=x^p\x=last(4>')} 



by definition of /, 



x tT ,t 



-fh,t + E E ^ fr ,« 

{(pefrags* (A)\4>=<j>'Tt} tr =x^p 



since /^ tr ( = when last (ft) ^ x 



-h,t + E E fx tr ,t 

tr=x^p {4>£frags'(A)\4>=<j>'Tt} 

=.fh,t + E fx tr ,t 
{tr—x-^p} 



by definition of / x (r ;t 



E 

ue{x|(x,t)e-E} 



by definition of _E 



case v e 5 \ {t} = V \ {A, ▼, t}: 

E = E E 

ue{i/|(u,i/)e£} tte{y|(t>^)e-E}0e/ra9s*(^i) 



by definition of f v<u 



<)!>e/ra 5 s*(.4) u£{!/|(i;,y)eB} 

= E I re + E ftv tr 

(pefrags* (A) \ {v tr \tr=v^p,last(</>)=v} 



by definition of by definition of ft r , 



El fv,[v]n E/ E/ /«,t) f ' 

0G/™gs*(.4) c/>£frags' (A) { v tr\ t r=v^pdast(4>)=v} 

E ^, t {c^mr) 

{cf>£frags* (A) | last (4>)=v} 

+ E E fiaAfyMMtr) 

(pefrags* (A) {tr=v^p\last(4>)=v} 



by definition of and of f*. 



E ^(^) a(<£)(±) + e ^x^) 

{0e/rags*(^t)|iast(0)=t)} \ {tr=v^p} 

E V<tA C 4>) 
{<f>£frags* (A)\last((j>)=v} 



since<7(</>)(r) = l-£ tr <7(<£)(ir) 



=M<7,t(C„) + E /w(Q>) 

{ipefrags'*(A)\4>=ip'Tv} 

=0+ E /^,t(<V) E (T(^)(*r)p(w) 

{0e/ra 9 s*(^)|0=0'rt)} {tr=ir^>p|iE=iast(0')} 



by definition of fJ, ai t(C^) 



{<t>efrags* (A)\4>=4>' tv} {tr=x ^ p\x=last{4>')} 
{4>efrags* (A)\4>=4>'Tt} {tr=x^-p\x=last(4>')} 



by definition of /J ((> 



E E ^x"> 

{^e/rags*(^)|0=0'Tt)} tr - x JL^. p 



since /^ (r = when last (ft) ^ x 



E/ E/ /a; tr ,t) 

tr=a;^p {<t>efrags* {A)\<t>=4>'rv} 

= E/ E/ ■£c tr ,w 

tr=x^p 4>' efrags' (A) 

= /x l, > 



by definition of f x tr 



= E /. 

ue{x\(x,v)£E} 

by definition of E 



case v £ 5 a : the proof is analogous. 

This concludes the proof that if there exists a scheduler a that induces an allowed weak transition 
t ==>c A*t such that p £(72) pt, then t =>c * £(^2) A* nas a solution /* (the flow / defined 
above) such that /<? T = p,(C) for each C £ 5/72. 

It is worth to note that for each state v, f Vb = Eae{*€^«*^)|tra«*(*)=6Afart(*)=«} AW(C«)- 
This property derives from the definition of /, the conservation of the flow constraints, and the 
definition of probability of cones. 

Since t =>c «• £(7?) A 4 nas a solution /*, it has also a solution f° that maximizes the objective 
function; since f° is a valid solution, it must satisfy the constraint v = /i(C) for each C £ 
5/7?., hence the statement if there exists a scheduler a for A that induces a weak transition 
t ==>c A 4 * such that \i £(7V) fit then t «• £(72) p has a solution f* such that /c - = M^) 
for each C £ 5/72 still holds. 
(=>) For a state x £ 5, let x be x if a = r and be x a if a ^ r. 

Given a solution /* of A ==>c * £(7?) A* such that T = /i(C) for each C £ 5/7?, define /x* as 
follows: for each state ieS, Pt(x) = /? and for each X C S, pt(X) = Exex A t t( a; )- 
It is straightforward to see that p t £ Disc(5): for each x, fit(x) = /? > and pt{S) — 

ExesM*) = Ex&sr /£,[«]* = Eees/K Exec ft,c = E C€ s/r/c,t = L The following 
property holds for p t : /i £(72.) /it- in f act , given an equivalence class C, pt{C) — Exec ^t( x ) = 
Exec /I C = /c ▼ = A*(C). The second equality follows from the definition of /i f while the last 
two equalities come from the constraints of t ==>q <^ £(7?) fi. 
Let g be a scheduler defined as follows: for each execution fragment <fi £ frags* (A), 

f* v tr/f* if /* ^ 0, trace(<f>) = e, and x = tr = v p E A; 

f*yr/f* if ^ 0, Arace(0) = e, a ^ r, and x = Ar = t> p £ A; 

fv a ,v^/fv a if /£, ^ °. *roce(^) = a ^ t, and x = tr = v -A p £ A; 

IfZ if jC* 7^ 0, trace((f>) =e,a = r, and a; = _L; 

*(<!>)(*) = { t Va , [v] Jtt a if it 7^ 0, Aroce(^) = a ± r, and x = J_; 

1 if trace((j)) £ {e, Arace(a)} and x = _L; 

1 if = 0, trace ((/>) = £ and a; = _L; 

1 if /* = 0, trace(<p) = a ^ r and a; = _L; 

otherwise 

where v — last(<f). 

It is interesting to observe that the above scheduler is a determinate scheduler since for each 

(/>, (j)' £ frags*(A) such that last(<f) = last(<p') and trace(<p) — trace(<f>'), we have cr(<fr) = 
c ((/>')■ In f act , given 0,0' £ frags* (A) such that last((f>) = last ((/)') — v and trace(<f>) = 
trace{4>'), if trace(<p) = trace ((/>') = e, then ct(0)(-L) = f*t v ] K /f£ = a-(<p')(±), for each 
transition tr = v — > p, a(<p)(tr) = = cr(0')(Ar), and for each transition Ar = 

v a((j))(tr) = f* v tr/f*, = (j{4>')(tr), as required. If trace(4>) = trace ((/>') = a ^ r, 

then cr(</>)(_!_) = /* r , //° = (t(0')(_L) and for each transition tr = v p, a{4>)(tr) — 



f* a , vl rlfv a = °{4>'){tr)\ for all other cases, either a(tf>){±) = 1 = <r(<f/)(±) or a(<f>)(x) = 
= a((j)')(x), thus for each 0,0' G frags*{A) such that last(cf)) = last{(j)') and trace((f>) — 
trace((f> r ), we have er(0) = <r(0'). 

Let He, j be the probabilistic execution fragment generated by a from t. In order to induce an 
allowed weak transition t ==>c pt, following conditions must be satisfied: 

1. for each G frags* (A), Supp(<r(0)) C A, 

2. n<y t t{frags*{A)) = 1, 

3. for each G frags*(A), if Mo\t(0) > then £race(0) = trace(a), and 

4. for each state i' G S, /U CT ,t({0 G frags*{A) \ lastQ>) = t'}) = p t (t'). 
We now prove that such conditions are actually satisfied: 

1. this follows immediately from the definition of a since for each transition tr such that 
a(4>){tr) >0,tre A, thus Supp(cr(0)) C A. 

2. Suppose that condition 4 holds. This implies that for each state v G S, /v,t({0 G 
frags* (A) \ last{4>) = v}) = p t (v), hence fi^ t (frags* (A)) = E„ e s /V,t({0 e 
frags* {A) \ last(4>) = v}) = E„ e <?Mt(f) = E„ e s = EceS/K E„ eC /l,C = 
Eces/K /c,t = L as required. 

3. Let be an execution fragment such that /v,t(0) > 0. Since /U CT ,t(0) = Pcr,t(C,p)a((f>)(±), 
Mcr,t( ( /') > holds if and only if p att (C^)a(<j))(±) > 0, that is, p a A^<t>) > an d 
cr(0)(-L) > 0. Now, assume that p a ,t(C^) > 0. According to the definition of the scheduler, 
cr(0)(l_) > holds if 

_ fv,[v]J?v > °> traced) = e, a = r, and v = lastly, 
~ fv a ,[v]Jfv a > °> ira ce(0) = a ^ r and v = last{4>)\ 

- trace{(f) £ {e, trace(a)}; 

- f* — 0, trace((f>) — e and x = _L; or 

- /* a = 0> trace{4>) = a ^ t and a; = _L; 

The first and last two cases imply that trace (0) = trace(a), as required; for the third case, 
we show that it can not occur if /^^(C^) > 0: suppose that trace(<p) £ {e, trace(a)}. This 
implies that trace{4>) = b for some sequence b of external actions with b ^ a. Denote by 
bi the first action of b and suppose that b\ ^ a. Let 0i and 02 be two execution fragments 
such that = 0i&i</>2 and trace(4>i) = £ and denote by vi and v 2 the last state of 0i and 
the first state of (f> 2 , respectively. The definition of probabilistic execution fragments and the 
fact that /v,t(C</>) > imply that PaA^^i) > 0' <J ( ( t ) i){t' r ) > and p(v 2 ) > for some 
transition tr = i>i — p. Since &i 7^ a and 61 7^ r, then by definition of the scheduler 

follows that a(<j>i)(tr) = for each transition tr = v\ p, thus /v,t(CV) = 0. This 
contradicts the hypothesis that /v,t(CV) > and hence trace{<p) {e, trace(a)} can not 
occur. If bi — a, consider b 2 and let 0i and <fr 2 be two execution fragments such that 
= <pib 2 <p 2 and trace{4>\) = a and denote by v\ and v 2 the last state of <f>\ and the first 
state of fa, respectively. The definition of probabilistic execution fragments and the fact that 
P<yA^<t>) > imply that /^((C^) > 0, a((p\)(tr) > and p(v 2 ) > for some transition 
tr = v\ — p. Since trace(tfii) = a 7^ r and 62 7^ t, then by definition of the scheduler 

follows that a(<j>i)(tr) = for each transition tr = Vi — ^> p, thus ^^^(C^) = 0. This 
contradicts the hypothesis that /v,t(CV) > and hence trace{<p) £ {e, trace(a)} can not 
occur. 

4. We first show by induction that for each x G S and each n G N, /* is an upper bound 
for the sum of the probabilities of the cones of execution fragments with empty trace and 
last state x within n steps, that is, denoted by F n (x) the set {0 G frags* (A) | trace{<p) = 
e, last{4>) = x, |0| < n}, E</, e F n (x) ^A c 4>) < fx'< similarly f* a is an upper bound for 
the sum of the probabilities of the cones of execution fragments with trace a ^ r and last 
state x within n steps, that is, denoted by F£(x) the set {<p G frags* (A) | trace{4>) = 
a, last(4>) = x, |0| < n}, E</)e-F a (x) ^<yA^<t>) — fx a - Note that for each » £ S and n G N, 
it holds that F n (v) C F„ +1 («) and F„» C 



We start showing that for each x G S and each n £ N, X)</>eF fx) Mo-,t(^) < fx'- 

Case n = and x = t: the only finite execution fragment tliat has length is <f> — t and 

this implies that E^F (t) ^A c <p) = ^A c t) = 1 = ft,t < it; 
Case n = and i/t: as in the previous case we have (j) = x, thus X^eF (a;) AV, *(</>) = 

MrriW = M^t(Cx) = < /*; 
Case n > and x = t: 

E V<r,t( C <)>) = Ma,t(C t ) + E V<j,t( C 4>'rt) 

<j>eF n (t) 4>'TteF n (t) 

= 1+ ^ /x<r,t(<V) X! <r(<f>')(tr)p(t) 

4>'TteF n (t) {tr=y^ r p\last(<j,')=y} 

= /:,*+E E E ^(^')(*r)p(t) 

veS^eFn^iy) tr=y^p 

= /a,* + E E E H*AC*>)<rW){tr) 

V&S tr=y JL+ p <t>'£.F n -i(y) 

= /:,*+E E E ^(c,') f -k^ 

= /:, + E E /»(*)% E 

</:, + E E pw^ 1 /; 
= /:,*+E E p(*)/vV 

f£S tr=y^^p 
= fh,t + E E ■A>"\* 

= fl,t + E ^* tr .* 

ir— z — s-p 

= h 

Case n > and x ± t: 

E MCT,t(C^) = Mc7,t(Cx) + E MCT,t(CVrx) 
0GF„(:c) (6'ti£F„(i) 

= E AV, t (CV) E <rW)(tr)p(x) 

<P'rxeF n (x) {tr=y^+p\last(4>')=y} 

= E E E 

= E E E H*AC*>M<t>'){tr) 

= E E E »«ACt>) f -*f 
= E E E 



<E E pw fJ ff; 

yeS tr =y^p J y 
= E E P( X )K,V» 

= E E fy tr ,* 

yeS tr=y^rp 

= E/ fz tr ,x 
tr—z^^p 

= /* 

This completes the proof that for each x e S and each n G N, E0eF„(iE) /wC^W < /x- 
Now we consider the second result relative to a ^ r, that is, for each ieS and each neN, 

Case n = 0: by definition of the trace of an execution fragment, we have that Fq(x) = 

and thus E^f^*) /V,t(C/>) = E<^0 /W^) = < 
Case n > 0: 

E M<r,t(C^) = E Mo-.^CVrx) + E A*(7,t(C^'ox) 
4>eF£(x) tj>'TxeF*(x) <p'axeF%(x) 

= E E 

4>'TxeF£(x) {tr=y^p\last(<p')=y} 
4>'axeF£(x) {tr=y^p\last(<p')=y} 

= E E ^,t(cv) E ^'x^m*) 

yeSpeF^y) tr=y^p 

+ E E ^,t(c o E <<t>')Wp(*) 

yeS 0'eF„_i(j/) tr= a ^p 

= E E E H*,t(C*>M4>')(tr) 

yeS tr=y j^ p <P'eFZ_ 1 ( y ) 

+ E E ^) E ^,t(<VM0')(*r) 

yeS tr=y ^ p 0'GF„_!(y) 

= E E />(*) E 
+ E E E 

= E E p^ J! fr~ E ^,t{ c <t>') 

y£S tr=y ^ p -ha 4>'eF^_ 1 ( y ) 

+ E E E 

yGS tr=s/ ^ p 0'eF„_i(y) 

^E E />(*)% £ & + E E 

yeS tr=y jL+ p yes tr=y _^ p h 



= E E p( x )fLvz + H E 

yes tr=y ^ p yes tr=y ^ p 

= E E fy%,xa + E E ^y?^a 
yes tr=y ^^ p yes tr=y ^^ p 

= E ^4 r ,^a + 51 ^W><, 

= /*. 

This completes the proof that for each x E S and each neN, E^eF"^) Mo\t(CV) < /^ . 
For each v E S, denote by the set U„ GN F n (u) and by i^ a (w)"the set \J neN F% (v): 

we have again that E^eFfz) AV,t(C ) < fx and E^eF-fz) V<?A c <i>) < fl a - Now k is 
immediate to show that for each state v E S, 

H<j,t{{4> G frags*(A) | Zost(^) = i>}) 

E /i„, t (C^)a(0)(±) 

{<p(zfrags* (A) \ last(<p)—v} 

= E ^, t (^)<7(0)(±) 

0GF(t))uF c '(«) 

+ E AV,t(C )a(0)(±) 

{0G/ra 9 s*(^)\(F(t))UF a (i;))|iast(0)=t)} 

= £ /^(cvmm-l) 

0eF(t))uF a (i;) 

0GF(«) 4>eF*(v) 

jt J 4>eF(v) Jv 

E VvAC'p) v Z [v]k otherwise 

>£F"(») A> 

- Ji} f* 

J y 

= M«) 

where the inequality is justified by the results about probabilities of cones we proved above 

and the equality = by the definition of the scheduler a that ensures that at least one between 
a(4>){±) and a(<j)')(±) is provided that 4> G F(v) and 0' E F a (v). So we have that for 
each?; E S, /v,t({</> G frags* {A) \ last{4>) — v}) < /Ut(u). 

Now, suppose for the sake of contradiction, that there exists a state v such that fJ, <Tl t(i < t > £ 
frags*(A) \ last((p) = v}) < Ht{v) and hence n ayt (frags* (A)) < 1 = ^t(S). This implies 
that there exists a set of infinite execution fragments E that occurs with non-zero probability. 
Since the set of states S is finite, there exists a set C C E and a state c (that can also be 
different from v) such that c occurs infinitely many times in each execution fragment E C 
and there exists a finite execution fragment (f> c with the following properties: 

- last((f> c ) = c; 

- C C C 0c ; 

- AV,t(U0ecCV,) = Ai CT: t(C0 c ); and 



- there exists a set L C frags* (A) such that <p c £ L, /^((U^lC^) — /x CT ,t(C^ ). an d 
for each <f> G L, <fi = (j) c biSi . . . b n s n for a family of actions bi and a family of states Si 
such that for each < i < n, ^ c and s„ = c. 
Denote by G the set {cb\Si . . . b n s n \ 3<p 6 L.<f> = (f) c bisi . . . b n s n }. Intuitively, the set G 
models the fact that from <\> c we enter in a cycle such that the probability to reach again c 
is 1 (and the probability to leave the cycle is 0) while the set L contains the finite execution 
fragments that extend cf) c by an execution fragment in G that can be seen as the generator 
of G, that is, it represents one loop of the cycle starting in c. Note that for each <p e G, 
trace ((f)) — e. Given an execution fragment such that last(cf)) = c, let (f>G n be the set of 
execution fragments defined as follows: 

j )G n = f{<f ) } ifn = 0and 

\ W<t>" I 4>' e , 4>" e G} if n > o. 

It is immediate to verify that L = § C G X and that for each i e N, M^tO-J^e^G*^) = 
fJ.a,t( c 4>J- Denote by 0G„ the set U o <i<„0G\ 

Now, suppose that a = t (the case a ^ ris analogous). Let k c be the length of (fi c , that 
is, fc c = \4>c\\ Pc be the probability of C < p c , that is, p c = /U CT ,t(C^ ); be the sum of the 
probabilities of the cones of length at most k c , that is, P c = J2<peF k (c) /wC^W' and Z\ c 
be /* - P c . Since /* is finite and p c > 0, I — \A c /p c ] + 1 is finite too; consider the 
set F(c) — U ne fiF n (c): by definition of the set F n (c) we have that for each < i < I, 
(t> c G l C F(c), thus 

E AV,t(G ) = E V*A C <t>) + E AV,t(G ) 

<t>eF(c)\(F kc (c)U<j> c G,) 

> X ^A c <p) + E ^A C 4>) 

= E E 

0<i<Z <f,e4>cG i 
0<i<Z 

= P C + ^ ^w(G^) 

0<j<2 
0<i<i 

= p c + Zp c 

= P c + ([Z\ c /p c ] +l)p c 
= P c + [Z\ c /p c > c +p c 

> "cH Pc + Pc 

Pc 

= P c + Zi c + p c 

= p c + f: -p c +pc 
= f:+p c 



but this contradicts the fact that X^gf(c) AV,t(C^) < /*; thus for each c G 5, p ai t({(f> G 
frags*(A) \ last((f>) — c}) — pt(c), as required. 

□ 

Corollary 2. Given a PA A an equivalence relation 72 on 5, an action a, a probability distribu- 
tion p G Disc(5), a set of allowed transitions A C D, and a state t G S, consider the problem 
t ==>c * £(72) A 4 as defined in Section\3\ 

t ==^c * £(72) A 4 ' 2fls a solution f* such that fc y ~ p{C) for each C G 5/72 if andonly if there 
exists a scheduler a for A that induces t =>q pt such that p £(72) pt such that for each state v, 

fv = J2aE{{lEfrags* (A)\last(0)=v} AV,i(Ca)- 

Proof. Given a scheduler a for .A that induces t ==>c A*t> by the proof of Theorem|2] we know that 
there exists a solution /* such that = p,(C) for each C G 5/7?. and such that for each state v, 

fv = Sae{,3e/ra 9 s*(.A)|!ast(/3)=i)} AV,t(Ca)- 

By the proof of Theorem [2] we know that given the optimal solution f° of the LP problem 
t ==>c o £(72) p, we can define a scheduler a inducing t =>c A 4 * sucn that A 4 £(72) /x* such 
that for each State q, T,<t,e{ a efrags*{A)\last(*)=q} V<rA C <i>) < fg- We claim that for each state 
1' T,^{ a Efrags'(A)\iast{ a )= q } P*A C 4>) = f°. Suppose, for the sake of contradiction, that there 
exists a state z such that E (t> e{ a efrags'{A)\iast(a)=z} ^A c 4>) < f°- Theorem implies that 
the LP problem t ==^q «■ £(7?) A 4 has a feasible solution /* such that for each state q, f* = 

Y.^e{ a efrags*(A)\last( a )= q } torA C 4>) ^ 4°' SillCe ft < f°< We haVe that maX T,(x,y)eE ~ fly < 

max E(i y )eE ~fx y hut this contradicts the fact that f° is an optimal solution. Hence it holds that 

Y,^{ a efrag S *(A)\iast( a )= q } HoA c *) = 4°> as required. □ 

Result 4 (CorollaryQ) Given a PA A, t G 5 and h £ S, a G S, p, p, p t & Disc(5), A C D, 
an equivalence relation 72 on S, the identity relation I on S Li {h}, a transition h — p, Ah = 
A U {h p}, D h = D U {h p}, and the PA Ah = (S U {h}, s, S, D h ), the following 
equivalences hold: 

1. t ==>c * £(72) A 4 ^ a ' s a solution f* such that j"c y = A 4 (^) for each C G 5/72 if and only if 
there exists a scheduler a for A inducing t =^c A 4 * such that p £(72) pt; 

2. h =^>Q h «■ £(72) p (h ==>q h <;> £(72) /xj relative to Ah has a solution f* such that fcj = 
p(C) for each C G 5/72 and only if there exists a scheduler a for A inducing p =>c pt 
(p ==>c A 4 *' respectively) such that p £(72) pt, 

Proof. The proof of the statement of the corollary involves Theorem[2]for the equivalence between 
the LP problem and allowed weak combined transition, Proposition [T] for ordinary transitions, and 
Proposition [2] for hyper- transitions . 
More precisely, 

1. the statement follows immediately from Theorem[2]and PropositionQ] 

2. By TheoremEl h ^>^ h * ^C^) A 4 has a solution /* such that T = p{C) for each C G 5/72 
if and only if there exists a scheduler ah for Ah that induces /i ==>Q h pt such that £(72) pt 
and the scheduler 07, exists, by Proposition |2] if and only if there exists a scheduler <r for ^4 that 
induces p =>q /i*. Since p t is reached also by a, p £(72) /it still holds, as required. The case 
for h =>Q h o £(72) p follows immediately by Proposition!]] 

□ 

Result 5 (Proposition |3) Given a PA A, two distributions pi, P2 G Disc(5), two actions ai, 0,2 G 
S, two sets A\^Ai C D of allowed transitions, and an equivalence relation 72 on S, the existence 
of pi,p2 G Disc(5) such that p\ =^q 1 pi, P2 ==^c 2 A*2» an d p\ £(72) p% can be checked in 
polynomial time. 



Proof. We remark that we denote by p =>q o £(11) p the problem h =>Q h o £(H) p relative to 
A h = (SU {h}, s,U,DU {h -A p}) where h $ S and A h = A U {h p}. 

Define the LP problem Pi 2 derived from the problems Pi = p\ : =>q 1 £(7t) p and P2 = 
pi ==> c 2 «> £(1Z) p as follows (after renaming of P2 variables to avoid collisions): the objective 
function of Pi 2 is the sum of the objective functions of Pi and P2; the set of constraints of Pi^ is 
J2ces/nPc = 1 together with pc > for C G S/1Z and the union of the sets of constraints of Pi 
and P2 where constraints fc.y = p(C) are replaced by fc.y = Pc- 

The proposition follows from the fact that P12 has a solution if and only if both Pi and P2 have 
a solution for some common probability distribution p and thus, by Corollary [Tj2j, if and only if 
pi and P2 enable an allowed hyper-transition to p\ and P2, respectively, such that p\ £(1Z) P2, as 
required, since p\ £(1Z) p as well as p 2 £(H) p and £(1Z) is transitive. It is immediate to see that 
Pi, 2 can still be generated and solved in polynomial time, since it is just the union of Pi and P2 
extended with at most N variables and 2N constraints where N = \S\. 

We now prove the above claim: 

Claim. Pi. 2 has a solution if and only if there exists a probability distribution p such that both Pi 
and P2 have a solution. 

- Pc 

Suppose that P12 has a solution and define p as follows: for each s 6 S, p(s) = — where 

1^ 

C — [s]tz. By hypothesis, Pi. 2 has a solution, that is, there exists /* that maximizes the objective 
function of Pi 2 while satisfying constraints. In particular, /* satisfies constraints: /* v > for 
each (u, v) e E; £\ c)g£ , f* c — f^y =0 for each C € S/1Z and seC; and T = p c for 
each C 6 S/1Z. Now, consider fl and obtained by splitting /* according to variables relative 
to Pi and P2, respectively. It is straightforward to check that /* is a valid solution for Pi with 
i = 1,2, so, by Corollary [TEJ, it holds that p\ £(1Z) p as well as P2 £(R) p 
(<^=) Suppose that there exists p such that both problems Pi = p\ ==>q 1 «> £(1Z) p and P2 = 
p2 =>q 2 * £(71) p have a solution. Suppose that the set of variables of P2 is disjoint from the 
set of variables of Pi. Let fl and the two solutions of Pi and P2 and denote by /* the union 
of /1 and /| extended with the assignments pc = p(C) for C € S/1Z. It is straightforward to 
check that /* satisfies all P12 constraints since they are just the union of constraints of Pi and 
P2 that are satisfied by /* and , respectively, and that the maximum of the objective function 
is given by /* since by definition the objective function is the sum of the two independent 
objective functions of Pi and P2 that are maximized by fl and , respectively. 



This concludes the proof of the claim and of the Proposition|3] 



□ 



